Method and system for providing scene data in a video stream

ABSTRACT

Methods of and systems for providing temperature data in a video stream are provided. The method includes receiving a video stream having a plurality of video frames with a first frame rate and receiving temperature data having a plurality of temperature frames with a slower second frame rate. To interlace the temperature data, a subset of temperature frames in the plurality of temperature frames can be extracted. The method further includes transmitting each temperature frame in the subset of temperature frames with the plurality of video frames in a data stream. The method may further include identifying missing data in the subset of temperature frames and correlating the missing data with the plurality of video frames. Based on the correlation of the missing data with the plurality of video frames, missing data can be provided to the subset of temperature frames to reconstruct the full plurality of temperature frames.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/785,856, filed Mar. 14, 2013 entitled “Method and System forProviding Scene Data in a Video Stream,” and U.S. Provisional PatentApplication No. 61/786,077, filed Mar. 14, 2013, entitled “Method andSystems for Producing a Temperature Map of a Scene,” the disclosures ofwhich are hereby incorporated in their entirety by reference for allpurposes. This application is related to U.S. patent application Ser.No. 14/211,796, filed on Mar. 14, 2014, entitled “METHOD OF SHUTTERLESSNON-UNIFORMITY CORRECTION FOR INFRARED IMAGERS.”

The following two regular U.S. patent applications (including this one)are being filed concurrently, and the entire disclosure of the otherapplication is incorporated by reference into this application for allpurposes:

Application Ser. No. 14/206,297, filed Mar. 12, 2014, entitled “Methodsand System for Producing a Temperature Map of a Scene”;

Application Ser. No. 14/206,341, filed Mar. 12, 2014, entitled “Methodand System for Providing Scene Data in a Video Stream”.

BACKGROUND OF THE INVENTION

The electromagnetic spectrum encompasses radiation from gamma rays,x-rays, ultra violet, a thin region of visible light, infrared,terahertz waves, microwaves, and radio waves, which are all related anddifferentiated in the length of their wave (wavelength). All objects, asa function of their temperatures, emit a certain amount of radiation.For terrestrial objects, a significant portion of this radiation isemitted in the infrared.

Thermal cameras can detect this radiation in a way similar to the way aphotographic camera detects visible light and captures it in aphotograph. Because thermal cameras detect and capture infrared light,thermal cameras can work in complete darkness, as ambient light levelsdo not matter and are not needed. Images from infrared cameras typicallyhave a single color channel because thermal cameras generally usesensors that do not distinguish different wavelengths of infraredradiation. Color thermal cameras require a more complex construction todifferentiate wavelength and color has less meaning outside of thenormal visible spectrum because the differing wavelengths do not mapuniformly into the system of color visible to and used by humans.

The monochromatic images from infrared cameras are often displayed inpseudo-color, where changes in color are used, as opposed to changes inintensity, to display changes in the signal, for example, gradients oftemperature. This is useful because although humans have much greaterdynamic range in intensity detection than color overall, the ability tosee fine intensity differences in bright areas is fairly limited.Therefore, for use in temperature measurement, the brightest (warmest)parts of the image are customarily colored white, intermediatetemperatures reds and yellows, transitioning to blues and greens, withthe dimmest (coolest) parts black. A scale should be shown next to afalse color image to relate colors to temperatures.

Thermal cameras have many applications, particularly when light andvisibility are low. For example, thermal cameras have been used inmilitary applications to locate human beings or other warm entities.Warm-blooded animals can also be monitored using thermographic imaging,especially nocturnal animals. Firefighters use thermal imaging to seethrough smoke, find people, and localize hotspots of fires. With thermalimaging, power line maintenance technicians locate overheating jointsand parts, a telltale sign of their failure, to eliminate potentialhazards. Where thermal insulation becomes faulty, building constructiontechnicians can see heat leaks to improve the efficiencies of cooling orheating air-conditioning. Thermal imaging cameras are also installed insome luxury cars to aid the driver at night. Cooled infrared cameras canbe found at major astronomy research telescopes, even those that are notinfrared telescopes.

SUMMARY OF THE INVENTION

Embodiments of the present invention generally relate to optical systemsand methods of processing thermal video signals. More particularly, thepresent invention relates to systems and methods of processing andtransmitting infrared scene data of a scene, such as sonar data (e.g.,for depth analysis) or thermal data (e.g., for temperature analysis).For example, a camera sensor may collect infrared scene data. The scenedata can be processed into infrared (IR) video, which may be suitablefor human viewing. The scene data may also be processed as temperaturedata which is useful for analyzing various aspects of the scene. Somescene data may not be suitable for human viewing, but may be used foranalysis in other processes. In other embodiments of the invention,cameras and other suitable systems may be used to detect and gatherscene data and video data of a scene, and transmit scene data in a videodata stream.

According to an embodiment of the invention, a radiometric infraredcamera is provided. The radiometric infrared camera can be configured todetect and capture scene data, such as thermal or sonar data, along withvideo data, of a scene. The video data may be processed separately fromthe scene data. For example, scene data such as thermal data capturedand processed as a two-dimensional matrix representing a temperature mapof the scene. The temperature map may then be transmitted separatelyfrom the video data to a camera interface for display to a user.

Video data may include visible light captured by the camera, andwavelengths of visible light captured typically may be in the range of400-700 nm. Thermal data may include non-visible light, e.g. infraredlight, captured by the camera, and wavelengths of infrared lightcaptured may be as long as 14 μm. The video data in one embodiment ofthe invention can come from a thermal sensor. The thermal sensor candetect emitted radiation in a long wave infrared (LWIR) spectrum, andmay not detect reflected radiation in the visible spectrum. The thermalsensor may operate in the LWIR spectrum (nominally 8-14 μm), and mayoutput thermal video. From the post-NUC (pre-AGC) thermal video, a LUTmay be used to create a temperature map. The camera then outputs a datastream that includes both the thermal video and the temperature map.

In one embodiment of the invention, the video contrast informationdescribed herein may be captured by a thermal camera. A change in thethermal radiation from an object in the scene may correspondsimultaneously to a change in both video contrast and temperatureinformation. The temperature information may be tracked at the same ordifferent precision and timescale that is important for viewing videocontrast information, and that updated temperature information may beprovided and refreshed to the system less frequently than video contrastinformation.

Accordingly, processing and transmitting video data and scene data(e.g., thermal data) at the same frequency and same range may bebandwith-intensive, utilizing heavy processing resources and memory, andincreasing processing time when a majority of the scene data may not beas relevant as the video data. However, other processing techniques ofscene data may compromise accuracy and completeness of the scene data.Embodiments of the invention address this and other problems, andprovide many benefits and advantages.

Methods of and systems for providing temperature data in a video streamare provided. The method includes receiving a video stream having aplurality of video frames with a first frame rate and receivingtemperature data having a plurality of temperature frames with a slowersecond frame rate. To interlace the temperature data, a subset oftemperature frames in the plurality of temperature frames can beextracted. The method further includes transmitting each temperatureframe in the subset of temperature frames with the plurality of videoframes in a data stream. The method may further include identifyingmissing data in the subset of temperature frames and correlating themissing data with the plurality of video frames. Based on thecorrelation of the missing data with the plurality of video frames,missing data can be provided to the subset of temperature frames toreconstruct the full plurality of temperature frames.

According to one embodiment of the invention, different processing andtransmission techniques may be used to to separately process andtransmit thermal data (e.g., a temperature map of a scene) and videodata of a scene. For example, since thermal data may vary in a smallerrelevant range and less quickly than video data, thermal data may becaptured in a full range but only relevant portions of the thermal datamay be processed and transmitted with the video data, withoutsignificantly compromising accuracy and completeness of thermal data ofthe scene. Selected portions of the thermal data may be transmitted withthe video data, decreasing the amount of total data being transmitted ina data stream, thus decreasing the bandwith needed.

According to one embodiment of the invention, a method of providingscene data in a video stream is provided. The method includes receivinga video stream having a plurality of video frames characterized by afirst frame rate. Subsequently, a scene data stream having a pluralityof scene data frames with a second frame rate is received. Each scenedata frame includes a plurality of scene data lines. The method furthercomprises generating a plurality of reduced scene data frames, whereineach of the plurality of reduced scene data frames is generated by: (a)extracting a subset of scene data lines from the plurality of scene datalines, wherein the subset is 1/N of the scene data lines; (b) forming aportion of a reduced scene data frame using the subset of scene datalines; and (c) iterating (a) and (b) N times. Once (a)-(c) have iteratedN times, then the method includes transmitting the plurality of videoframes at the first frame rate and the plurality of reduced scene dataframes at the second frame rate in a data stream.

The method of providing scene data according to an embodiment of theinvention may further include receiving the data stream and separatingthe plurality of video frames from the plurality of reduced scene dataframes. For each reduced scene data frame in the plurality of reducedscene data frames, the method includes: identifying one or more missingdata lines in the subset of scene data lines, correlating the pluralityof reduced scene data frames with the plurality of video frames,providing the one or more missing data lines to the subset of scene datalines of the reduced scene data frame based on the correlation of theplurality of reduced scene data frames with the plurality of videoframes, and creating a reconstructed scene data frame including the oneor more missing data lines and the subset of scene data lines of thereduced scene data frame. Then a plurality of reconstructed scene dataframes is generated using the reconstructed scene data frame created foreach reduced scene data frame and the plurality of reconstructed scenedata frames is transmitted.

In another embodiment of the invention, a system for providing scenedata in a video stream is provided. The system includes a detectorconfigured to receive a video stream having a plurality of video frameswith a first frame rate and receive a scene data stream having aplurality of scene data frames with a second frame rate, each scene dataframe including a plurality of scene data lines. A frame processingmodule may be configured to, for each scene data frame in the pluralityof scene data frames: extract a subset of scene data lines in theplurality of scene data lines, thereby interlacing the scene datastream, create a reduced scene data frame including the subset of scenedata lines, and generate a plurality of reduced scene data frames usingthe reduced scene data frame created for each scene data frame. Thesystem may further comprise a first transmitter configured to transmitthe plurality of reduced scene data frames at the second frame rate withthe plurality of video frames at the first frame rate in a data stream.

The system according to an embodiment of the invention may furthercomprise a receiver configured to receive the data stream and separatethe plurality of video frames from the plurality of reduced scene dataframes. A frame reconstructing module in the system may be configuredto, for each reduced scene data frame in the plurality of reduced scenedata frames: identify one or more missing data lines in the subset ofscene data lines, correlate the plurality of reduced scene data frameswith the plurality of video frames, provide the one or more missing datalines to the subset of scene data lines of the reduced scene data framebased on the correlation of the plurality of reduced scene data frameswith the plurality of video frames, create a reconstructed scene dataframe including the one or more missing data lines and the subset ofscene data lines of the reduced scene data frame, and generate aplurality of reconstructed scene data frames using the reconstructedscene data frame created for each reduced scene data frame. Then asecond transmitter in the system may be configured to transmit theplurality of reconstructed scene data frames.

Reconstructing complete thermal data may be performed after transmissionof the selected portions of the temperature data and the reconstructionmay be based on correlations between the video data and thermal data ina post-process of the thermal data. Thus, benefits and advantagesachieved by embodiments of the present invention include reducedbandwidth without compromising accuracy and completeness of thermaldata, or other scene data. These and other embodiments of the invention,along with many of its advantages and features, are described in moredetail in conjunction with the text below and attached figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an example system for transmittingtemperature data in a video stream according to an embodiment of theinvention.

FIG. 2 shows an example of temperature data input and temperature dataoutput illustrating an example method of transmitting temperature datain a video stream according to an embodiment of the invention.

FIG. 3 shows an example of temperature data input and temperature dataoutput illustrating an example method of reconstructing the transmittedtemperature data from the video stream according to an embodiment of theinvention.

FIG. 4 is a flowchart diagram illustrating an example method ofreconstructing the temperature data according to an embodiment of theinvention.

FIG. 5 shows a block diagram of an example system for transmittingtemperature data in a video stream according to an embodiment of theinvention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Embodiments of the invention are related to methods and systemsconfigured to capture, process, and transmit video and scene data. Inone example, radiometric infrared cameras can capture video data as wellas thermal data. In an embodiment of the invention, a portion of thescene data, such as thermal data (e.g., temperature map of the scene),may be selected and transmitted in a data stream with full video data.The thermal data and video data are analyzed and based on correlationsbetween the partial thermal data and the full video data, missingthermal data is recreated based on the correlations. Thus, the missingthermal data, with the partial thermal data transmitted, may be used toreconstruct full thermal data. Accordingly, the thermal data may betransmitted at a lower rate than the video data, and may require lesscomputing, processing, and storage resources related to transmittinglower amounts of thermal data relative to the full video data beingtransmitted.

Embodiments of the invention provide for methods of providing atemperature map with video data. In an embodiment of the invention, thetemperature data is transmitted at a reduced frame rate using an N-fielddata interlacer, which splits the temperature data over multiple fieldsto be transmitted in portions. The split temperature data is thentransmitted as part of the video codec frame in a data stream via a datastream transmitter. The video frame with a portion of the temperaturemap is received by a data stream receiver, via a USB, UWB, Wi-Fi, or anyother suitable wired or wireless connection or communication protocol.

The video data from the data stream is extracted by and into a videocodec, and the temperature data from the data stream is extracted by ade-interlacer, which uses the video data to fill in the missing gaps intemperature data. The temperature data in some embodiments contains gapsbecause it was transmitted at a significantly lower bit rate and thusonly portions of the temperature data were transmitted with the videodata. The separated video data and temperature data is then communicatedto a display processor such that the video data and temperature data maybe displayed on various devices (e.g., a smartphone, tablet, orcomputer). Displaying the video data with the temperature data allows anend user to interact with the device to determine the temperature ofdifferent scenes captured in the video data.

FIG. 1 shows a block diagram illustrating a high-level example system100 for transmitting scene data in a video stream according to oneembodiment of the invention. System 100 illustrates an example of aradiometric infrared camera enabled to capture thermal (e.g.,temperature) data and video data for transmission in a video streamaccording to one embodiment of the invention. In other embodiments ofthe invention, other devices and systems may be used such that othertypes of scene data may be captured, processed, and transmitted; forexample, sonar data may be used to provide depth analysis of a scene. Inthe example shown in FIG. 1, thermal data may be captured by aradiometric infrared camera 118.

In one embodiment of the invention, the camera 118 may capture thermaldata and convert the thermal data into infrared (IR) video, usingprocessing techniques such as automatic gain control (AGC), local areaprocessing (LAP), etc. The thermal data may then be used to generate atemperature map (e.g., radiometric data). The IR video data is processedwith a video codec 102 and then transmitted by a data stream transmitter106, in some embodiments, to a display side, such as a monitor, phone,or other external device. In another embodiment, the radiometric camera118 may be enabled to process and display thermal and video data inreal-time on a monitor or display on the body of the radiometric camera.To display the IR video data on display 116, such as a monitor ortelevision, the video data in the data stream may be received at a datastream receiver 108. The IR video data in the data stream is processedagain through a video codec 110 on the display side, which is thenprocessed through a display processor 114 and the video data may beviewed on display 116.

In an embodiment of the invention, there may be a sensor processingmodule (not shown) between the camera 118 and the video codec102/N-field scene data interlacer 104. The sensor processing module mayreceive sensor data gathered from the camera and generate an outputincluding a video stream and radiometric data (e.g., thermal). The videostream may then be processed through the video codec 102, and theradiometric data may be processed through the N-field scene datainterlacer 104.

In another embodiment of the invention, the camera may be configuredsuch that it uses a visible light sensor and same or differenttechniques to minimize data transmissions. The camera 118 maysimultaneously capture visible light video of a scene along with thethermal data of the scene.

Simultaneously, thermal data may be captured by the camera 118. To havethe video data and thermal data correlate properly, the thermal data maybe transmitted at the same frame rate as the video data because thethermal data may be used to numerate temperature on the display 116. Forexample, to show on the display a ball bouncing across the floor, thethermal data may show the temperature that is representative of the ballas it is moving. Thus the thermal data may be in synchronization withthe video data to accurately represent the temperature of the scene inreal-time.

According to one embodiment of the invention, scene data, such asthermal data, may be processed using an N-field interlacer 104. Incontrast to visible light video (e.g., television) typically processedin two fields such that odd lines are processed first and the even linesnext, thermal data may be processed using N-fields where N can be anumber of two or greater, and can be an odd number. Thermal data may beprocessed with more than two fields because it may be processed at aslower rate than video data. Visible light video data (e.g., televisionvideo) can be processed in two fields to capture more data quickly, forexample, television video data representing moving objects. Televisionvideo data may be sent to the display 116 at a high frame rate, but canbe compressed for transmission between the data stream transmitter 106and the data stream received 108. In contrast, thermal data may betransmitted at a lower frame rate than the video data. However, toretain accuracy, different compression algorithms may be used totransmit thermal data from compression algorithms used to transmit videodata.

Historically, this has been done as a way to save video bandwidth. Ourembodiment extends that to radiometric data and also extends by goingbeyond 2 fields. In an embodiment of the invention, a full video frameand 1/nth of the radiometric frame is transmitted.

FIG. 2 illustrates a thermal input 202 of an N-field interlacer 104 ofFIG. 1 and its resulting thermal output 208. For example, in a sequenceof four frames representing four-field processing (i.e., N=4), eachframe has a plurality of lines containing a full thermal data of ascene. Frame 1 (204) has lines 1 through line h, up to Frame 4 (206),also having lines 1 through line h. Intermediate frames 2 and 3 (notshown) also may have lines 1 through line h containing full thermal dataof the scene per frame. In the embodiment shown in FIG. 2, a frame is arepresents thermal data that has been captured over a unit of time suchas 16.67 ms. In another embodiment of the invention, a frame can be ashutter capture of thermal data and the lines that frame contain thermaldata that may be transmitted at a frequency (e.g., frame rate).

A field may represent a subset of lines from the frame. Thus a 4-fieldinterlacer 104 would transmit for a first frame (e.g., field), only(Line MOD 4=1) lines are transmitted. MOD is the mathematical modulusoperator representing the remainder of a divisional operation. Thereforefor the input frame 1 (204) of the interlacer input 202 containing linesLine 1 through Line h, only lines with line numbers having 1 as aremainder after being divided by 4 (e.g., 1, 5, . . . , h-3) would betransmitted, resulting in a first output frame 210 containing lines Line1 and other lines from Line 5 through Line h-3 having MOD 4=1. For asecond frame (not shown) having lines Line 1 through Line h, only lines(Line MOD 4=2) are transmitted; therefore, the interlacer output 208 forthe second frame can be shown in the second output frame 212 containinglines Line 2 and other lines from Line 6 through Line h-2 having MOD4=2. For a third input frame (not shown) having lines Line 1 throughLine h, only lines (Line MOD 4=3) are transmitted.

Accordingly, the interlacer output 208 for the third frame (not shown inFIG. 2) can be represented by the third output frame 214, containinglines Line 3 and other lines from Line 7 through Line h-1 having MOD4=3. For a fourth frame 206, only lines (Line MOD 4=0) are transmitted;therefore the interlacer output 208 for the fourth frame 206 can berepresented by the fourth output frame 216, containing lines Line 4 andother lines from Line 8 through Line h having MOD 4=0. The interlacer104 may continue for further frames in the plurality of framescontaining thermal data.

In an embodiment shown in FIG. 2, first frame 204 may contain a completeset of temperature data, from which every 4th line is extracted fortransmission and outputted into the interlacer output 208. Therefore,upon the receipt of 4 frames in the interlacer output 208 (shown asframes 210, 212, 214, and 216), a complete set of temperature data forthe first frame is received. Accordingly, frame 216 of the interlaceroutput 208 contains line 4 of a frame in the interlacer input 202 (e.g.,from frame 1 204 or frame 4 206). In this embodiment the temperature mapis only used every 4 frames, therefore a single temperature map is usedfor all 4 frames. The unused temperature data is discarded.

In another embodiment, the method can include storing all of the data atthe first frame. In the 4th cycle, line 4 in 216 is created with aportion of the temperature data is extracted from the originaltemperature map. Accordingly, line 4 of 216 is temperature data threecycles later than the temperature data in line 1 in 210 of theinterlacer output 208. As such, every frame contains a full temperaturemap, but only a portion of the temperature map is extracted,transmitted, and reconstructed on the receiving end.

By processing the thermal data input 202 using the N-field interlacer104 of FIG. 1, the bandwidth required to transmit the thermal data maybe reduced, as less lines per frame of thermal data are sent to the datastream transmitter 106 of FIG. 1. The video data may continue processingat a video frame rate and all frames are transmitted with all lines ofvideo data to the data stream transmitter 106. Thus, the N-fieldinterlacer 104 is used for selecting a subset of lines of frames forscene data, such as thermal data and/or sonar data, to determinetemperature and/or depth of a scene, respectively. Therefore, accordingto an embodiment of the invention, the bandwidth for scene data (e.g.,thermal data) may be reduced by a factor of N relative to the video databandwidth. The bandwidth for scene data may be enabled to transmit thesame number of frames originally received in the interlacer input 202 ofFIG. 2, but for each interlacer input frame having a total number oflines of data, a subset of the lines of data are selected to create areduced frame in the interlacer output 208, each reduced framerepresentative of each original frame having the total number of lines.As such, a plurality of reduced scene data frames is generated by theN-field interlacer 104 to create interlaced scene data.

Referring back to FIG. 1, the data stream transmitter 106 can transmit adata stream including a full video stream from the video codec 102 andinterlaced scene data stream from the interlacer output 208 of FIG. 2 asrepresenting the analytical (e.g., thermal) data. Accordingly, in orderto display the full video stream with a full analytical (e.g., thermal)data stream, the full scene data stream can be reconstructed based onthe interlaced scene data stream from the interlacer output 208, whichcan be a reduced scene data stream since it includes a plurality ofreduced frames having a portion or subset of the data lines originallyin the full frame. To reconstruct the plurality of scene data frameshaving the total number of lines of data in each frame, the full videodata may be analyzed. A correlation may be determined between the fullvideo data and the reduced scene data. For example, thermal data andvideo data can have high correlation properties related to movement.

For example, a most recent video frame corresponding to a reducedthermal frame may be used to make some intelligent decisions on how torecreate missing data from the reduced thermal frame to reconstruct thefull thermal frame. In one embodiment of the invention, each thermalframe can include a temperature map, thus only portions of thetemperature map may be outputted from the N-Field interlacer 104 andtransmitted by the data stream transmitter 106. The data stream received108 may separate the reduced thermal data (e.g., portions of temperaturemap for each frame) and provide the reduced thermal data to a scene datade-interlacer 112. The scene data de-interlacer 112 can receive areduced thermal frame, e.g. portions of a temperature map for everyframe, collecting multiple reduced thermal frames until a full thermalframe can be reconstructed.

The scene data de-interlacer 112 uses the video data to interpolate thetemperature map to missing data in the temperature map. When there ismotion in the scene, the motion may be used, for example in the videodata, to gather more recent temperature data. Alternatively, when thereis no motion, different types of interpolation may be used to fill inthe missing data in the temperature map.

In an embodiment of the invention, the N value for N-field interlacingthe scene data may be adjustable with a boundary condition. For example,for N=9, a system may not have even multiples of nine lines of dataavailable. So if a frame has 24 lines long, additional lines containingvalues of zeros may be added to create a frame of 27 lines for N-fieldinterlacing with N=9.

The data stream transmitter 106 of FIG. 1 may transmit the data streamincluding video data and interlaced scene data stream using varioustechniques and methods. According to one embodiment of the invention,packetization of the data may occur to transmit the data stream from thedata stream transmitter 106 to the data stream receiver 108. The camera118 can be configured to transmit video data from the camera 118 to anexternal device (e.g., a phone) via a communications interfaces.Exemplary communications interfaces may include a Wi-Fi link forwireless communication or an USB link, thus transmission of data mayinclude packetized segments of the data. Thus a full frame of video dataplus a portion of the frame of scene data (e.g., thermal data) ispacketized for transmission. For example, 70 packets can equal one totalsuper frame, in which a super frame includes a full frame of video dataand a full frame of scene data. The packets are transmitted by the datastream transmitter 106 and received at the external device, such as aphone. The received packets are then stored to build a buffer in thephone of total super data. Therefore for every one frame of video, aportion of one frame of scene data is transmitted. For example,referring to FIG. 2, for each frame of video data, one fourth of a frameof thermal data is transmitted. So over four frames of video data, oneframe of thermal data is received, thus after the transmission of 4 fullframes of video data, a complete frame of thermal data is received,representing a full temperature map of the scene.

FIG. 2 shows an example of an interlacer input 202 and its correspondinginterlacer output 208. As shown in FIG. 2, the first frame of theinterlacer input (Frame 1 (204)), includes frame lines line 1 throughline h. Intermediate frames (i.e., Frame 2 and Frame 3) are not shown inthis diagram for purposes of clarity. Frame 4 (206), which is the fourthframe of the interlacer output 202, includes frame line 1 through lineh. As illustrated, each of the interlacer input frames includes allframe lines, i.e., lines 1 through line h.

The interlacer output 208 shows the output frames 210, 212, 214, and 216corresponding to each input frame (i.e., Frame 1 204, Frame 2 (notshown), Frame 3 (not shown), and Frame 4 206, respectively. For example,frame 210 of the interlacer output 208 is the output corresponding toinput frame 204 of the interlacer input 202. As can be observed in theoutput frames 210, 212, 214, and 216 of the interlacer output 208, eachoutput frame contains every fourth line of the corresponding inputframe. Thus, in each output frame, one quarter of the lines in thecorresponding input frame are utilized to form the interlacer output outframes.

The interlacer 104 of FIG. 1 uses the interlacer input 202, whichcontains all of the video data to generate the interlacer output 208,which contains a selected portion (e.g., one quarter) of the data fromthe input. The portioned data is combined into a superframe with thevideo data, which is transmitted in the data stream transmitter 106 ofFIG. 1. Although the N-field interlacer is shown in FIG. 2 to have N=4,any number for N may be used to divide the lines of the temperature datainto portions.

According to one embodiment of the invention, during packetization, thepackets containing the scene data are different from the packets ofvideo data. For example, the scene data may have a different header fromthe video data, since the scene data and video data are processeddifferently and originate from different sources.

In other embodiments of the invention, transmission of the data streamfrom the data stream transmitter 106 to the data stream receiver 108 mayoccur through other methods and techniques of communication. Forexample, a wired (e.g., USB) or a wireless (e.g., Wi-Fi,telecommunications) communication interface may be utilized.

As an exemplary implementation, for cases in which the video data ishighly correlated with the temperature data, a cross correlation woulddemonstrate the similarities between video and temperature data. In thiscase, a higher resolution could be utilized for portions of the frame ofvideo data to interpolate the temperature at higher resolution. For lesscorrelated regions, existing data could be utilized to interpolate thetemperature at lower resolution.

FIG. 3 illustrates a de-interlacer input 302 for the scene datade-interlacer 112 of FIG. 1. The interlaced scene data stream,containing reduced scene data frames may be received as thede-interlacer input 302. The reduced scene data frames (e.g., reducedthermal data frames) may be stored in a buffer of an external device,such as a phone, monitor, or television. The de-interlacer input 302 maybe reconstructed with missing data to create reconstructed data 312. Forexample, missing data may be provided and inserted between differentlines that composed the reduced scene data frame received in thede-interlacer input 302.

For example, the de-interlacer input 302 can have four fields (e.g.,frames) corresponding to four frames of the interlacer output 208 ofFIG. 2. The de-interlacer input 302 of FIG. 3 may include Field 0 (304),Field 1 (306), Field 2 (308), and Field 3 (310), corresponding tointerlacer output 208 of FIG. 2, such as output frame 1 (210), outputframe 2 (212), output frame 3 (214), and output frame 4 (216),respectively. Field 0 (304) may include Line 1 and other lines from Line5 through Line h-3 having MOD 4=1, corresponding to a first output frame210 of FIG. 2. Field 1 (306) may include lines Line 2 and other linesfrom Line 6 through Line h-2 having MOD 4=2, corresponding to a secondoutput frame 212 of FIG. 2. Field 2 (308) may include lines Line 3 andother lines from Line 7 through Line h-1 having MOD 4=3, correspondingto a third output frame 214 of FIG. 2. Field 3 (310) can include linesLine 4 and other lines from Line 8 through Line h having MOD 4=0,corresponding to a fourth output frame 216 of FIG. 2.

To provide the missing data lines in the reconstructed data 312, thevideo data may be analyzed to determine a correlation between the videodata and the scene data (e.g., thermal data). For example, the videodata and/or scene data (e.g., thermal data) can be highly correlated.Having knowledge of the content of the video data, for example, thelocation of objects are in that video and motion detected from theobjects in the video, it can be determined as to which lines of theframes of thermal data are relevant and which lines are not. Forexample, in the analysis of the video data, if there is no motionbetween Field 0 and Field 3 in the video data, then any of the lines inthe thermal data may be used since they are most likely to be unchanged(e.g., the same or insignificantly different). However, if there ismotion, then the lines of thermal data occurring most recently to themotion detected in the video data may be selected to most accuratelyreflect that motion. More accurately, when there is motion, the missinglines are computed using an interpolation algorithm and using the datafrom the last good lines in the most recent field.

According to one embodiment of the invention, initially complete thermaldata, such as a full temperature map, may be collected at the same pointand time as full video data was collected. A portion of the thermaldata, such as a portion of the temperature map, can be transmitted witheach frame of the video data, such that eventually a full temperaturemap is eventually constructed after a certain number of video dataframes are collected. As the full temperature map is being built withportions of the temperature map taken at different points and times, thetotal temperature map may have artifacts in it that are associated withrapid motion in the full video data. Therefore the motion defined by thevideo data processed by the video codec may help determine if the sceneis a still scene, and either add all the partial thermal data frames,separate the frame of the video data at a time point, or select all ofthe thermal data frames from the original or last frame in the sequenceat the time point. Missing data may be generated using an interpolationalgorithm to compensate for motion in how the de-interlacer input 302 issupplemented to create the reconstructed data, for example, insertingmissing data to rebuild the temperature map.

In one embodiment of the invention, the de-interlacer can fill inmissing data shown in the re-construction data 312 of FIG. 3 using datafrom previous N fields. However, when there is motion in the image fromone field to another, older fields lines may no longer be valid for thatregion. The de-interlacer makes use of the high correlation between thetemperature stream and the video stream. In a region where thecorrelation is high, there may be no motion and the de-interlacer mayuse old line data to fill in the missing data lines. In regions wherecorrelation is low, there may be motion, therefore the most recent fieldlines may be used in an interpolation algorithm to create the missingdata to be used to fill in the missing data lines. Many interpolationalgorithms may be used to create the missing data lines. In oneembodiment of the invention, linear interpolation algorithms may beused. In another embodiment of the invention, quadratic interpolationalgorithms may be used.

Corresponding to Field 0 (304) of the de-interlacer input 302, areconstructed first frame can be shown in 314, having missing data linesin between the transmitted data lines Line 1 and Line 5 through Line h-3having MOD 4=1. A reconstructed second frame can be shown in 316,corresponding to Field 1 (306), can have missing data lines in betweenthe transmitted data lines Line 2 and Line 6 through Line h-2 having MOD4=2. Corresponding to Field 2 (308), a reconstructed third frame can beshown in 316 and may have missing data lines in between the transmitteddata lines Line 3 and Line 7 through Line h-1 having MOD 4=3. Areconstructed fourth frame 320, corresponding to Field 3 (310), can havemissing data lines in between the transmitted data lines Line 4 and Line8 thought Line h having MOD 4=0.

Using the reconstructed data frames 314, 316, 318, and 320, theoriginally transmitted lines may be extrapolated to recreate a fullthermal data frame 324 as a de-interlacer output 322. The full thermaldata frame 324 may have lines Line 1 to Line h.

As shown in FIG. 3, the de-interlacer output 322 is the output of thescene data de-interlacer 112 of FIG. 1. The reconstruction frames in 312represent an intermediate step that is performed by the de-interlacer.The reconstruction process involves processing line 1, field 0, anddetecting missing data between line 1 and line 5. Initially, the missingdata may be filled with zeros, for processing purposes. Thereconstruction process further includes reconstructing the frame withthe missing data, which is done by processing video data and withrespect to motion or other data in the video, interpolate the missingdata. For example, interpolating missing data into the fields containingzeros may be based on information in the video data and information inthe existing temperature map to generate actual lines 1 through h shownin the output frame 324 of the de-interlacer output 322.

If there is no motion in the scene, frame 324 of the de-interlaceroutput 322 would include lines from a combination of reconstructionframes 314, 316, 318 and 320. When there is no motion in the scene, thenthe lines of data from frame to frame have little variation, so copyingthe lines from one frame to another does not degrade the accuracy of thedata significantly. However, when there is motion, then there isvariance in the lines of data from frame to frame in the reconstructionframes 312. Therefore, the de-interlacer 112 of FIG. 1 interpolates thedata in frame 320 because the most recent data is desired to representthe temperature. The reconstruction involves interpolating between thelines 4, 8, etc. of frame 320 to generate a temperature map and may useas part of that motion information some correlated properties betweenthe video and the temperature map.

When the data is transmitted, the scene data and the video data directlycorrespond to each frame one to one. For example, when a new frame ofvideo data is received, a full, new frame of scene data or temperaturedata is received at the same time. The bandwidth is reduced fortransmission by interlacing is because at the destination the samebandwidth is not needed as the temperature does not change as quickly.Therefore, if the end user requests a high response then the N-fieldinterlacer may be set to a lower N value. Alternatively, a higherN-value for the N-field interlacer results in a slower response or lowerbandwidth for the scene data. The effect of changing N values is thatthe number of fields may effect the time it takes to reconstruct a fullframe (e.g., more fields results in more cycles to receive every 1/Nthframe to reconstruct a full frame).

In an embodiment of the invention, the thermal camera may utilize abuffer to store frames of scene data that are being received at aconstant rate. The frames of scene data may be stored and the retrievedfrom the buffer as appropriate. For example, when a new frame of scenedata is received, a portion of interlaced data for the frame may betransmitted and then combined with scene data from the buffer toreconstruct the whole frame of scene data.

FIG. 4 shows a flow diagram illustrating an example algorithm 400 for amethod of de-interlacing represented in FIG. 3 and performed by thescene data de-interlacer 112 of FIG. 1. The method 400 begins at step402 as the start, where scene data is received from the data streamtransmitter 106 of FIG. 1. The scene data received may be represented bythe interlacer output 208 of FIG. 2 or the de-interlacer input 302 ofFIG. 3. At step 404, a parameter “field_counter” may initially be set to0. The de-interlacer 112 of FIG. 1 may then receive and store a field ofscene data (e.g., thermal data) in step 406. The scene data may bestored and processed in a field storage RAM in step 408. In step 416,the “field_counter” parameter may be incremented by 1, and it is checkedin step 418 whether it has reached N (e.g., N=4 in FIGS. 2 and 3).

If “No”, the de-interlacer 112 repeats steps 406 and 416 to receive andstore another field of scene data. If “yes”, the de-interlacer 112fetches video data in step 420. Video data processed by the video datacodec in step 424, is fetched in step 420. Analyzing the video data, itis determined whether there is motion in the video in step 422. If thereis no motion (i.e., “No”), in step 414 the fields of scene data storedin the field storage RAM are interpolated with missing lines from olderfields of scene data. However, if there is motion (i.e., “Yes”), in step410 the fields of scene data are interpolated with missing lines fromthe most recent field data. Then the missing data lines from step 410(e.g., recent field data representing motion) and the missing data linesfrom step 414 (e.g., previously stored data representing lack of motion)can be used to build an output scene data map in step 412. Then themethod returns to step 404 to reset the “field_counter” restart at 0.

In another embodiment, as an alternative to interpolation, the missingdata may be filled at the destination. A buffer may temporarily befilled with the lines that were coming in based on the frame count.Therefore, knowing what the frame number is or the frame count expectedand the input video data, the de-interlacer may determine where to placethe lines to create data to fill in for the missing data duringreconstruction. If a packet is dropped, the buffer is not cleared outunless there is a rest of the entire system, as old data stored in thebuffer may still be utilized in displaying a new temperature map.

Referring to FIG. 4, initially the field counter may be set to zero inreconstructing temperature data. On the reconstruction side so, thede-interlacer receives and stores a field of scene data 406, processedin RAM 408, and increments the counter. The de-interlacer receives aquarter of the field of scene data at a time.

When the counter reaches the N value at 418, the process moves to fetchnew video data at 420, which allows use of the most recent video data.The de-interlacer may then determine if there is any motion in the sceneat 422 and use that to determine how to interpolate the data. Motiondata may be extrapolated from the most recent video data fetched in 420,or alternatively extrapolated from older video data, as shown at 414.The motion data is used in 410 and 412 to build a full scene data map.To achieve lower data rates, in an embodiment, interpolation may beskipped altogether because the temperature does not change significantlyenough to compensate for the processing time of interpolatingtemperature data constantly.

Furthermore, in transmitting data over a wireless network (e.g., wi-fi),a significant amount of data may be lost. Therefore in an embodiment ofthe invention, if there motion in the video, then the de-interlacerinterpolates the missing lines from the most recent field data. Forexample, if at the 4th cycle (where N=4), the de-interlacer fetches thevideo data, and motion is detected, then the de-interlacer fills in thetemperatures for lines 1, 2 and 3 based on the temperature in the 4thframe, including the 4th line. Subsequently, in order to generate the5th line, the de-interlacer interpolates 4 and 8 of what is stored inthe field storage RAM at 408. 4th and 8th lines. And you will interpretthe other lines from that.

If on the other hand there is no motion then, the de-interlacer may usestored in the field storage RAM, because it is full after N cycles. Thefield storage RAM may store all the temperature maps, for which theentire map may be used as received.

In another embodiment, there can be an intermediate stage where there isa little motion, which may be approached similarly as in noisereduction. For example, the interlacer or de-interlacer may utilize acertain percentage, or a greater percentage of the most recent data overwhat was in the other ones. The percentage of data from the more recentvide data may be weighted toward the most recent frame. As such, asliding scale for proportions of recent video data may be implementedwhen the is little motion to interpolate some of the lines, however asignificant amount of data may still be extrapolated from the storeddata in the field storage RAM.

It should be appreciated that the specific steps illustrated in FIG. 4provide a particular method of receiving partial scene data (e.g.,thermal data) and reconstructing the partial scene data to result infull scene data (e.g., temperature data of a scene represented by atemperature map). Other sequences of steps may also be performedaccording to alternative embodiments. For example, alternativeembodiments of the present invention may perform the steps outlinedabove in a different order. Moreover, the individual steps illustratedin FIG. 4 may include multiple sub-steps that may be performed invarious sequences as appropriate to the individual step. Furthermore,additional steps may be added or removed depending on the particularapplications. One of ordinary skill in the art would recognize manyvariations, modifications, and alternatives.

In one embodiment of the invention, scene data may be displayed at aslower frame rate than video data, for example, thermal data, such as atemperature map, may be displayed at a slower rate than the video dataintentionally. Video data may be displayed at a rate to representreal-time, for example, 9 Hz to 30 Hz. However, a measurement cycle forthe thermal data may be much lower. Therefore, a sample of thetemperature data, such as a portion or a fraction of the temperaturedata, may be transmitted for every frame of video data. As a result,after a certain number of frames of video data have been transmitted, afull temperate map is eventually collected from the fractions oftemperature data transmitted with each frame. For example, if one fourthof the temperature map is transmitted with each video frame, then afterfour video frames, a full temperature map can be received. Advantagesachieved by transmitting fractions of the temperature data with eachframe of video data include a reduced bandwidth since less data is beingtransmitted.

In another embodiment of the invention, a temperature map may betransmitted at a frame rate that is a fraction of the video frame rate.The video data may have a refresh rate of 9 Hz to 120 Hz (e.g., 9 Hz to30 Hz), but the temperature data may have a lower refresh rate, forexample 1 Hz to 120 Hz (e.g., 1 Hz to 5 Hz). In an exampleimplementation using a 9/1 Hz ratio, the temperature map is shown everyninth frame of video data. However, in other embodiments, a greaterbandwidth of the communication medium may be used to implement the fulltemperature map at rates closer to the video data frame rate. Forexample, the temperature refresh rate may range up to 30 Hz, or be atthe same rate as the video refresh rate. Thus, embodiments of thepresent invention include a variety of frame rates for the video data(e.g., 9 Hz, 30 Hz, 60 Hz, 120 Hz, or the like) and a variety of framerates for the video data (e.g., 1 Hz, 5 Hz, 9 Hz, 30 Hz, 60 Hz, 120 Hz,or the like). It should be noted that embodiments of the presentinvention are not limited to these particular frame rates.

In other embodiments of the invention, the video data and thermal datamay be gathered by a device other than an infrared camera. Othersuitable devices can include a visible light camera that is coupled withan acoustical imaging system or infrared imaging system.

In another embodiment of the invention, scene data may includeacoustical data, which may be translated into a depth map of a scene.

In another embodiment of the invention, scene data can be sent at afraction of the rate of the frame rate of the video data. Obtaining thevideo data may occur at a pre-determined frame rate. Thus, thetransmitting a portion of the scene data may occur at a frame rate lessthan the pre-determined frame rate of the video data.

FIG. 5 illustrates an exemplary system 500, such as a video and scenedata processing system, enabled to execute the functions and processesdescribed above. The system may comprise a processing module 516, suchas a central processing unit, or other computing module for processingdata. The system 500 may include a non-transitory computer-readablemedium 502, such as a static or dynamic memory (e.g., read-accessmemory, or the like), storing code for executing tasks and processesdescribed herein. For example, the computer-readable medium may comprisea video codec process 506 to process video data received. Scene data maybe processed and interlaced by the N-field scene data interlacer 504.The data stream of combined video data from the video codec processed bythe video codec processor 506 and the output from the N-field scene datainterlacer 504 may be transmitted by data stream transmitter 514.

The computer-readable medium 502 may further comprise a data streamreceiver 512 to receive the combined data stream. The scene data may beprocessed by an scene data de-interlacer 510. To reconstruct the missingdata, a video and scene data analyzer 508 may access scene data framedatabase 518 and video frames database 520 via the processor 516 todetermine what analytic frames to use as missing data lines. The missingdata lines are used by the scene data de-interlacer 510 to reconstructthe scene data from the interlacer 504 output. The reconstructed scenedata may be transmitted to external interfaces 524 (e.g., monitor,phone) through an I/O module 522, or other suitable interface module,via various communications methods. Example communication methods may bewired or wireless, using hardware and software components, such as a USBinterface, Wi-Fi interface, and/or Bluetooth interface or other suitablecommunications interface.

FIG. 5 is a high level schematic diagram illustrating a data processingsystem upon which the disclosed embodiments may be implemented incertain embodiments. Embodiments may be practiced with various computersystem configurations such as infrared cameras, hand-held devices,microprocessor systems, microprocessor-based or programmable userelectronics, minicomputers, mainframe computers and the like. Theembodiments can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network. FIG. 5 shows one example of adata processing system, such as data processing system 500, which may beused with the present described embodiments. Note that while FIG. 5illustrates various components of a data processing system, it is notintended to represent any particular architecture or manner ofinterconnecting the components as such details are not germane to thetechniques described herein. It will also be appreciated that networkcomputers and other data processing systems which have fewer componentsor perhaps more components may also be used. The data processing systemof FIG. 5 may, for example, a personal computer (PC), workstation,tablet, smartphone or other hand-held wireless device, or any devicehaving similar functionality.

For example, the data processing system 500 can include a system buswhich is coupled to a microprocessor, a Read-Only Memory (ROM), avolatile Random Access Memory (RAM), as well as other nonvolatilememory. The microprocessor may be coupled to a cache memory. System buscan be adapted to interconnect these various components together andalso interconnect components to a display controller and display device,and to peripheral devices such as input/output (“I/O”) devices. Types ofI/O devices can include keyboards, modems, network interfaces, printers,scanners, video cameras, or other devices well known in the art.Typically, I/O devices are coupled to the system bus through I/Ocontrollers. In one embodiment the I/O controller can include aUniversal Serial Bus (“USB”) adapter for controlling USB peripherals orother type of bus adapter.

RAM can be implemented as dynamic RAM (“DRAM”) which requires powercontinually in order to refresh or maintain the data in the memory. Theother nonvolatile memory can be a magnetic hard drive, magnetic opticaldrive, optical drive, DVD RAM, or other type of memory system thatmaintains data after power is removed from the system. While thenonvolatile memory can be a local device coupled with the rest of thecomponents in the data processing system, it will be appreciated byskilled artisans that the described techniques may use a nonvolatilememory remote from the system, such as a network storage device coupledwith the data processing system through a network interface such as amodem or Ethernet interface (not shown).

With these embodiments in mind, it will be apparent from thisdescription that aspects of the described techniques may be embodied, atleast in part, in software, hardware, firmware, or any combinationthereof. It should also be understood that embodiments can employvarious computer-implemented functions involving data stored in a dataprocessing system. That is, the techniques may be carried out in acomputer or other data processing system in response executing sequencesof instructions stored in memory. In various embodiments, hardwiredcircuitry may be used independently, or in combination with softwareinstructions, to implement these techniques. For instance, the describedfunctionality may be performed by specific hardware componentscontaining hardwired logic for performing operations, or by anycombination of custom hardware components and programmed computercomponents. The techniques described herein are not limited to anyspecific combination of hardware circuitry and software.

Embodiments herein may also be in the form of computer code stored on acomputer-readable medium. Computer-readable media can also be adapted tostore computer instructions, which when executed by a computer or otherdata processing system, such as data processing system 500, are adaptedto cause the system to perform operations according to the techniquesdescribed herein. Computer-readable media can include any mechanism thatstores information in a form accessible by a data processing device suchas a computer, network device, tablet, smartphone, or any device havingsimilar functionality. Examples of computer-readable media include anytype of tangible article of manufacture capable of storing informationthereon such as a hard drive, floppy disk, DVD, CD-ROM, magnetic-opticaldisk, ROM, RAM, EPROM, EEPROM, flash memory and equivalents thereto, amagnetic or optical card, or any type of media suitable for storingelectronic data. Computer-readable media can also be distributed over anetwork-coupled computer system, which can be stored or executed in adistributed fashion.

Alternatively, a field-programmable gate array (FPGA) integratedcircuit, application specific integrated circuit (ASIC), any suitableconfigurable hardware and/or any suitable configurable integratedcircuit may be used to implement one or more of the functions andprocesses of the embodiments described herein.

It is also understood that the examples and embodiments described hereinare for illustrative purposes only and that various modifications orchanges in light thereof will be suggested to persons skilled in the artand are to be included within the spirit and purview of this applicationand scope of the appended claims.

What is claimed is:
 1. A method of providing scene data in a videostream, the method comprising: receiving a video stream having aplurality of video frames characterized by a first frame rate; receivinga scene data stream including a thermal data stream, the scene datastream having a plurality of scene data frames with a second frame rate,each scene data frame including a plurality of scene data lines;generating a plurality of reduced scene data frames, wherein each of theplurality of reduced scene data frames is generated by: (a) extracting asubset of scene data lines from the plurality of scene data lines,wherein the subset is 1/N of the scene data lines; (b) forming a portionof a reduced scene data frame using the subset of scene data lines; and(c) iterating (a) and (b) N times; and transmitting the plurality ofvideo frames at the first frame rate and the plurality of reduced scenedata frames at the second frame rate in a data stream.
 2. The method ofclaim 1 wherein the second frame rate is lower than the first framerate.
 3. The method of claim 1 wherein the thermal data stream comprisesa temperature map associated with the video stream.
 4. The method ofclaim 1 wherein the scene data further comprises sonar data.
 5. Themethod of claim 1 wherein the first frame rate and the second frame rateare a same frame rate.
 6. The method of claim 1 wherein the first framerate ranges from 9 Hz to 120 Hz and the second frame rate ranges from 1Hz to 120 Hz.
 7. The method of claim 1, further comprising: receivingthe data stream; separating the plurality of video frames from theplurality of reduced scene data frames; for each reduced scene dataframe in the plurality of reduced scene data frames: identifying one ormore missing data lines in the subset of scene data lines, correlatingthe plurality of reduced scene data frames with the plurality of videoframes, providing the one or more missing data lines to the subset ofscene data lines of the reduced scene data frame based on thecorrelation of the plurality of reduced scene data frames with theplurality of video frames, and creating a reconstructed scene data frameincluding the one or more missing data lines and the subset of scenedata lines of the reduced scene data frame; and generating a pluralityof reconstructed scene data frames using the reconstructed scene dataframe created for each reduced scene data frame; and transmitting theplurality of reconstructed scene data frames.
 8. The method of claim 7,further comprising transmitting the plurality of reconstructed scenedata frames with the plurality of video data frames to an externaldisplay.
 9. The method of claim 7 wherein the second frame rate is lowerthan the first frame rate.
 10. The method of claim 7 wherein the datastream includes a plurality of packets, each packet including onereduced scene data frame in the plurality of reduced scene data framesand a subset of video frames in the plurality of video frames.
 11. Themethod of claim 10 wherein the reduced scene data frame is reduced by afraction inversely proportional to a number of video frames in thesubset of video frames.
 12. A system for providing scene data in a videostream, comprising: a detector configured to receive a video streamhaving a plurality of video frames with a first frame rate and receive ascene data stream including a thermal data stream, the scene data streamhaving a plurality of scene data frames with a second frame rate, eachscene data frame including a plurality of scene data lines; a frameprocessing module configured to, for each scene data frame in theplurality of scene data frames: extract a subset of scene data lines inthe plurality of scene data lines, thereby interlacing the scene datastream, create a reduced scene data frame including the subset of scenedata lines, and generate a plurality of reduced scene data frames usingthe reduced scene data frame created for each scene data frame; and afirst transmitter configured to transmit the plurality of reduced scenedata frames at the second frame rate with the plurality of video framesat the first frame rate in a data stream.
 13. The system of claim 12wherein the first frame rate and the second frame rate are a same framerate.
 14. The system of claim 12 wherein the thermal data streamcomprises a temperature map associated with the video stream.
 15. Thesystem of claim 12 wherein the scene data further comprises sonar data.16. The system of claim 12, further comprising: a receiver configured toreceive the data stream and separate the plurality of video frames fromthe plurality of reduced scene data frames; a frame reconstructingmodule configured to, for each reduced scene data frame in the pluralityof reduced scene data frames: identify one or more missing data lines inthe subset of scene data lines, correlate the plurality of reduced scenedata frames with the plurality of video frames, provide the one or moremissing data lines to the subset of scene data lines of the reducedscene data frame based on the correlation of the plurality of reducedscene data frames with the plurality of video frames, create areconstructed scene data frame including the one or more missing datalines and the subset of scene data lines of the reduced scene dataframe, and generate a plurality of reconstructed scene data frames usingthe reconstructed scene data frame created for each reduced scene dataframe; and a second transmitter configured to transmit the plurality ofreconstructed scene data frames.
 17. The system of claim 16 wherein thesecond transmitter is configured to transmit the plurality ofreconstructed scene data frames with the plurality of video data framesto an external display.
 18. The system of claim 16 wherein the secondframe rate is lower than the first frame rate.
 19. The system of claim16 wherein the data stream includes a plurality of packets, each packetincluding one reduced scene data frame in the plurality of reduced scenedata frames and a subset of video frames in the plurality of videoframes.
 20. The method of claim 19 wherein the reduced scene data frameis reduced by a fraction inversely proportional to a number of videoframes in the subset of video frames in each packet.